Data Cleaning for Word Alignment
نویسنده
چکیده
Parallel corpora are made by human beings. However, as an MT system is an aggregation of state-of-the-art NLP technologies without any intervention of human beings, it is unavoidable that quite a few sentence pairs are beyond its analysis and that will therefore not contribute to the system. Furthermore, they in turn may act against our objectives to make the overall performance worse. Possible unfavorable items are n : m mapping objects, such as paraphrases, non-literal translations, and multiword expressions. This paper presents a pre-processing method which detects such unfavorable items before supplying them to the word aligner under the assumption that their frequency is low, such as below 5 percent. We show an improvement of Bleu score from 28.0 to 31.4 in English-Spanish and from 16.9 to 22.1 in German-English.
منابع مشابه
Word Alignment Based Parallel Corpora Evaluation and Cleaning Using Machine Learning Techniques
This paper presents a method for cleaning and evaluating parallel corpora using word alignments and machine learning algorithms. It is based on the assumption that parallel sentences have many word alignments while non-parallel sentences have few or none. We show that it is possible to build an automatic classifier, which identifies most of non-parallel sentences in a parallel corpus. This meth...
متن کاملAutomatic Thesaurus Generation through Multiple Filtering
11, this paper, we propose a method of gen(',rating bilingual keyword eh.lsters or thesauri from parallel or comi.m, able bilingual corpora. The method combines nmrphological and lexical processing, bilingual word aligmnent, and graph-theoretic cluster generation. An experiment shows that the method is promising. 1 I n t r o d u c t i o n In this paper, we propose a method of automatte bilingua...
متن کاملLattice Score Based Data Cleaning For Phrase-Based Statistical Machine Translation
Statistical machine translation relies heavily on parallel corpora to train its models for translation tasks. While more and more bilingual corpora are readily available, the quality of the sentence pairs should be taken into consideration. This paper presents a novel lattice score-based data cleaning method to select proper sentence pairs from the ones extracted from a bilingual corpus by the ...
متن کاملSpeech recognition for medical conversations
In this paper we document our experiences with developing speech recognition for medical transcription – a system that automatically transcribes doctor-patient conversations. Towards this goal, we built a system along two different methodological lines – a Connectionist Temporal Classification (CTC) phoneme based model and a Listen Attend and Spell (LAS) grapheme based model. To train these mod...
متن کاملData Issues in English-to-Hindi Machine Translation
Statistical machine translation to morphologically richer languages is a challenging task and more so if the source and target languages differ in word order. Current state-of-the-art MT systems thus deliver mediocre results. Adding more parallel data often helps improve the results; if it doesn’t, it may be caused by various problems such as different domains, bad alignment or noise in the new...
متن کامل